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Aviation  antisubmarine  warfare  operators 

Acoustic  analysis 

Sonar  simulators 

lob  performance  testing 


^?he  objective  of  this  research  was  to  determine  whether  the  test  currently  used  to 
measure  proficiency  In  acoustic  analysis,  which  differs  substantially  from  the  OL-S2  sonar 
system  used  in  the  fleet,  provides  an  accurate  estimate  of  operator  on-the-job  perfor¬ 
mance.  Subjects  (22  AWs  enrolled  In  the  Air  Anti-Submarine  Squadron  Forty-one  vVS-*l) 
operator  course)  were  administered  six  single  target  grams  to  analyze  and  classify  on  the 
DL-82  sonar  system  simulator;  and  six,  using  static  lofargrams.  Measures  of  acoustic 
analysis  (identification  of  sound  source,  predominant  spectrum,  and  propeller  mode),  and- 
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classification  (accurate  identification  of  the  target)  decisions  were  obtained  under  both 
testing  situations. 


3  Subject  performance  on  the  analysis  variables  differed  significantly  for  the  two 
display  formats.  Subjects  more  accurately  identified  predominant  spectrum,  sound 
source,  and  propeller  mode  on  the  OL-S2  sonar  system  simulator  than  they  did  on  the 
static  linear  iofargrams.  Subjects'  performance  on  the  classification  task  was  not 
significantly  different  for  the  two  display  formats.  It  was  concluded  that  operator 
proficiency  on  the  OL-82  sonar  system  simulator,  at  least  for  AWs  in  the  training 
pipeline,  cannot  be  estimated  accurately  using  static  linear  iofargrams. 


S/N  0102*  L*  014*  MSI 


FOREWORD 


This  research  and  development  was  performed  in  response  to  recognized  Navy  needs 
for  an  investigation  of  the  effects  of  initial  training  and  job  conditions  on  skill  retention 
under  subproject  ZF63-522-001-010  (Computer-aided  and  Classroom  Training),  work  unit 
03.07  (Skill  and  Knowledge  Retention).  In  previous  work  conducted  to  investigate 
relationships,  problems  were  observed  in  the  consistency  between  different  performance 
measures. 

The  objectives  of  the  subproject  are  to  (1)  derive  ways  of  detecting  potential 
problems  existing  in  the  Navy  and  (2)  recommend  means  to  minimize  performance 
deterioration  by  restructuring  training  and  job  conditions.  The  objective  of  this  effort 
was  to  examine  the  relationship  between  the  assessment  of  operator  performance  in  a 
high-fidelity  simulator  and  the  use  of  proficiency  tests  that  are  of  relatively  low  fidelity 
to  the  actual  performance  situation  to  estimate  operational  readiness.  Results  are 
intended  primarily  for  the  Anti-Submarine  Warfare  Wing,  Pacific,  and  for  other  agencies 
concerned  with  aviation  sonar  operator  training  and  testing.  This  report  should  also  be  of 
interest  to  agencies  responsible  for  the  development  of  skill  maintenance  programs  and 
proficiency  tests  for  personnel  in  technical  ratings. 

Appreciation  is  expressed  to  members  of  the  instructor  staff  at  the  Air  Anti- 
Submarine  Squadron  Forty-one  (VS-41),  Naval  Air  Station,  North  Island,  who  were 
instrumental  in  facilitating  the  gathering  of  data  and  who  provided  advice  on  technical 
matters.  Particular  appreciation  is  expressed  to  AWCS  Sather  for  his  support  throughout 
the  research  project. 


JAMES  F.  KELLY,  JR.  JAMES  W.  TWEEDDALE 

Commanding  Officer  Technical  Director 
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Aviation  antisubmarine  warfare  operator*  (AWs)  in  the  $-3A  community  are  tasked 
with  classifying  targets  using  acoustic  analysis  techniques.  The  task  is  accomplished 
onboard  die  aircraft  using  dynamic,  computer-based,  passive  acoustic  sonar  equipment 
(OL-&2  sonar "  system).  Acoustic  information  is  presented  on  logarithmically-scaled 
iofargrams  using  'formats  on  a  CRT  screen.  'Multiple  job  aids  are 

available  f or  problem solution. 

In  contrast,  the operational  readiness  of  theee  operators  is  estimated  based  on  results 
of  a  test  using  static  iofargrams  to  present  acoustic  infortfetion.  Sob  aids  available  to 
the  operators  in  these  tests  are  limited  to  mechanicai  dividers  and  hand-held  ealeukrtors. 

Objective 


The  objective  of  tills  research  was  to  determine  whether  the  currently  used 
proficiency  test,  which  differs  substantially  from  the  OL-S2  sonar  system,  both  in  the 
form  of  presentation  and  in  support  for  tite  analysis  procedures,  provides  an  accurate 
estimate  of  operator  on-the-job  performance. 

Approach 

Subjects  were  22  students  enrolled  in  tite  Air  Anti-SUbmarine  Squadron  Forty-one 
(VS-tl)  operator  course.  The  subjects  were  administered  six  single  tenet  grams  to 
analyze  and  classify  an  the  OL-S2  sonar  system  simulator  and  six,  using  static  iofargrams. 
The  grams  used  for  the  two  test  administrations  were  equated  ter  level  of  difficulty  by 
site ject  matter  experts  using  tite  Ititigrsnt  tifftcuffy  index*  Measures  of  acoustic  analysis 
(identification  of  sound  source,  predominant  spectrum,  and  propaUer  mode),  and 
classification  (accurate  identification  of  the  tar  got)  decisions  were  obtained  under  both 
testing  situations. 

Findints 

Subject  performance  on  tite  analysis  variables  Offered  significantly  for  the  two 
display  formats.  Subjects  mere  accurately  identified  predominant  spectrum,  sound 
source,  and  ptepefler  mode  on  the  OL-S2  sonar  system  simulator  than  they  dbf  on  the 
static  linear  iofargrams.  Subjects'  performance  on  the  classification  task  was  not 
significantly  Afferent  for  the  two  display  formats.  Correlations  between  the  two  dtapta y 
formats  for  analysis  and  classification  scores  were  untfotmly  lew.  Analysis  significantly 
correlated  with  classification  scores  for  performance  on  tite  OL-*2  sonar  system 
iKmufsAte  but  for’  performance  on  static  kfanrems. 
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INTRODUCTION 


Problem  and  Background 

Operation  of  complex  systems  in  the  modern  Navy  requires  that  technical  personnel 
maintain  a  high  level  of  skill.  Proficiency  tests  are  administered  to  personnel  to  assess 
their  level  of  technical  competence  and  to  make  estimates  of  operational  readiness. 
However,  constraints  or  limitations  on  resources  for  testing,  such  as  cost,  administrative 
difficulties,  and  unavailability  of  operational  equipment  at  test  sites,  may  necessitate 
compromises  in  testing  that  reduce  the  accuracy  of  the  proficiency  tests  in  determining 
operational  readiness  in  fleet  operators/technicians.  In  addition,  because  of  these  testing 
constraints,  the  skills  tested  by  the  proficiency  tests  may  be  significantly  different  than 
those  performed  on  the  job.  As  a  result,  personnel  may  have  to  practice  and  maintain 
skills  specifically  for  the  test,  which  would  actually  distract  their  attention  from 
maintaining  the  skills  used  on  die  job. 

There  is  considerable  recognition  of  the  problems  associated  with  accurate  measure¬ 
ment  of  job  performance.  Vineberg  and  Joyner  (1982)  point  out  that  the  development  and 
administration  of  job  sample  performance  tests  are  difficult,  time  consuming,  and  often 
do  not  yield  objective  and  meaningful  measures  of  performance.  Osborn  and  Ford  (1976) 
and  Ellis,  Wulfeck,  and  Fredericks  (1979)  stress  the  requirement  for  a  high  degree  of 
similarity  between  the  actual  task  and  how  it  is  measured.  Ellis  et  al.  (1979)  developed 
procedures  for  assessing  the  adequacy  of  performance  tests,  emphasizing  die  need  for 
consistency  between  the  test  standards,  conditions,  and  actual  task  performance  on  the 
job. 


Aviation  antisubmarine  warfare  operators  (AWs)  in  the  Navy's  S-3A  community  are 
tasked  with  classifying  targets  using  computer-based,  passive  acoustic  sonar  equipment 
(OL-82  sonar  system).  Acoustic  information  is  presented  on  logarithmically-scaled 
lofargrams  on  a  CRT  screen  using  multiple-display  formats.  Several  job  aids,  including 
computational  programs  and  electronic  dividers,  are  available  for  problem  solution. 

In  contrast,  for  skill  maintenance  training  and  proficiency  testing  in  the  fleet,  a  quite 
different  form  of  the  task  is  used.  For  example,  in  the  S-3A  squadrons  at  Naval  Air 
Station,  North  Island,  AWs  practice  on  static  lofargrams  to  maintain  their  acoustic 
analysis  and  classification  skills.  These  skills  are  tested  on  a  quarterly  proficiency  exam 
that  requires  the  operator  to  analyze  and  classify  static  lofargrams.  The  Wing  Command 
then  interprets  the  scores  obtained  to  estimate  the  proficiency  of  those  AWs  on  the  OL-82 
sonar  system  used  in  the  S-3A  aircraft.  However,  operators  in  those  squadrons  reported 
that  they  spent  about  twice  as  much  time  per  week  studying  static  lofargrams  for  the 
quarterly  wing  exam  than  they  spent  performing  the  task  in  the  aircraft  (Konoske,  Wetzel, 
&  Montague,  1983). 

Currently,  the  proficiency  tests  must  present  acoustic  information  on  single-display, 
static,  linear-scaled  lofargrams  because  neither  the  OL-82  sonar  system  in  die  aircraft 
nor  the  OL-82  simulator  used  in  the  operator  training  course  is  available  for  proficiency 
testing.  Additionally,  the  lofargram  display  on  the  OL-82  sonar  system  is  produced  with  a 
logarithmic  measurement  scale.  At  this  time,  mechanical  logarithmic  dividers  do  not 
exist.  Since  the  lines  on  the  gram  cannot  be  measured  without  such  a  device,  analysis 
procedures  cannot  be  accomplished  on  printed  copies  of  OL-82  displays.  The  only  job  aids 
available  to  the  operator  in  the  test  conditions  are  linear-scaled  mechanical  dividers  and 
hand-held  calculators,  neither  of  which  are  used  by  the  operator  while  performing  the  job 
under  operational  conditions. 
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Although  it  appears  that  the  knowledge  and  skills  required  to  analyze  acoustic  data 
on-the-job  and  from  printed  lofargrams  are  similar,  the  two  tasks  are  not  "the  same" 
because  the  AW  using  the  OL-82  sonar  system  has  a  greater  number  and  types  of  job  aids 
available  to  help  perform  acoustic  analysis  than  does  the  AW  using  the  printed  lofargram 
proficiency  test.  The  accuracy  of  using  operator  performance  on  static,  linear  scale 
lofargrams  to  estimate  operational  readiness  on  a  logarithmically  scaled,  computer-based 
sonar  system  needs  to  be  assessed. 

Objective 

The  objective  of  this  research  is  to  obtain  data  on  operator  performance  of  acoustic 
analysis  and  classification  skills  on  the  OL-82  sonar  system  simulator  and  the  lofargram 
proficiency  tests.  Such  data  will  be  used  to  determine  whether  scores  on  the  static 
lofargram  are  accurate  estimates  of  performance  on  the  OL-82  sonar  system. 


APPROACH 


Subjects 

Subjects  were  22  students  enrolled  in  the  Air  Anti-Submarine  Squadron  Forty-one 
(VS-41)  operator  course.  All  had  attended  the  Common  Core  Acoustic  Analysis  School, 
Fleet  Aviation  Specialized  Training  Group,  prior  to  attending  the  VS-41  operator  course. 

Test  Materials/Procedures 


In  all  cases,  subjects  were  administered  the  OL-82  exam  prior  to  the  static  lofargram 
test.  The  order  of  the  tests  was  a  condition  required  by  the  VS-41  instructors. 

OL-82  Sonar  System  Simulator 

The  final  exam  for  unit  four  (lofargram  analysis)  of  the  VS-41  operator  course  was 
used  for  the  computer-based  sonar  system  performance  test.  Six  single-target  grams 
were  presented  serially  to  subjects  seated  in  the  OL-82  positional  trainer.  Job  aids,  such 
as  aural  cues  (presented  through  earphones  worn  by  subjects),  electronic  dividers,  track 
ball,  and  computational  programs,  were  available  for  subject  use.  Display  options 
included  Broadband,  Vernier,  Translate,  ALI,  and  Demon  presentations. 

Subjects  reported  analysis  and  classification  decisions  over  a  communications  phone 
to  an  instructor  positioned  outside.  The  instructor  evaluated  the  subject's  performance  on 
the  variables  of  interest  using  a  predetermined  scoring  checklist. 

Static  Linear  Lofargram 

Six  single-target  grams  were  presented  on  static  linear  lofargrams  to  subjects  seated 
at  tables  in  a  classroom  setting.  These  grams,  which  were  photocopies  of  grams  produced 
on  the  fast  analyzer  system  (FAS),  had  been  judged  by  subject  matter  experts  to  be  of 
equal  difficulty  to  those  used  in  the  OL-82  sonar  system  simulator  test.  Job  aids  available 
to  the  subjects  were  limited  to  mechanical  dividers  and  hand-held  calculators.  Subjects 
wrote  analysis  and  classification  decisions  on  an  answer  sheet  provided  for  the  test. 
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Test  Scores 


1.  Gram  analysis  performance  scores  were  accurate  identification  of  predominant 
spectrum  (PS),  sound  source  (SS),  and  propelier  mode  (PM)  of  the  target  presented  on  the 
grams  on  both  displays. 

2.  The  classification  score  was  accurate  identification  of  targets  for  grams  (A) 
presented  on  both  displays. 

Analyses 


1.  An  analysis  of  covariance  was  performed  on  the  analysis  and  classification  scores 
for  the  two  display  modes,  using  subject's  armed  forces  qualification  test  (AFQT)  score  as 
the  covariate. 

2.  Mean  percentage  scores  and  standard  deviations  (SDs)  for  gram  analysis  and 
classification  on  the  two  display  types  were  computed. 

3.  Correlations  were  computed  for  and  between  gram  analyses  and  classification 
scores  obtained  using  the  two  displays. 


FINDINGS 

Table  1,  which  presents  results  of  the  analysis  of  covariance  performed  on  the  scores 
obtained  using  the  two  display  modes,  shows  that  test  performance  due  to  display/system 
differences  differed  significantly  for  the  analysis  variables  but  not  for  classification 
accuracy. 


Table  1 

Analysis  of  Covariance  Summary  Table  for  the  Effect  of  Display 
on  Analysis  and  Classification  Scores 


Score 

df 

MS 

F 

Analysis: 

Sound  source 

1 

22095.4 

103.60** 

Error 

21 

203.4 

Predominant  spectrum 

1 

22230.0 

59.20** 

Error 

21 

375.1 

Propulsion  mode 

1 

1276.6 

4.16* 

Error 

21 

306.8 

Classification: 

Accuracy 

1 

56.8 

.18 

Error 

21 

315.1 

*p  <  .03. 

**p  <  .01. 
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Table  2,  which  presents  mean  percentage  scores  and  SDs  for  scores  obtained  on  the 
two  display  types,  shows  that  the  SS  and  PS  analysis  scores  obtained  by  subjects  on  the 
static  lofargram  test  were  about  half  those  obtained  on  the  OL-S2  sonar  system  test. 
However,  on  the  classification  score,  subjects  scored  somewhat  higher  on  the  static 
lofargram  test  than  on  the  OL-82  test. 


Table  2 

Mean  Percentages  and  Standard  Deviations  for  Gram  Analysis 
and  Classification  Scores  Using  Both  Types  of  Displays 


Score  OL-82  Simulator  Static  Lofargram 

Mean  SD  Mean  SD 


Analysis 


Sound  source  (SS) 

93.04 

10.03 

48.22 

19.80 

Predominant  spectrum  (PS) 

93.81 

9.87 

48.86 

24.83 

Propulsion  mode  (PM) 

80.77 

18.18 

70.00 

22.88 

Classification 

Accuracy  (A) 

80.81 

19.56 

83.09 

17.21 

Table  3  presents  analysis  and  classification  score  intercorrelations  for  both  types  of 
display.  As  shown,  analysis  scores  were  significantly  correlated  with  classification  scores 
for  targets  displayed  on  the  OL-82  sonar  system  simulator,  but  not  for  targets  displayed 
on  the  printed  lofargrams.  Finally,  Table  4  shows  that  correlations  between  scores 
obtained  using  the  two  displays  were  uniformly  low  and  nonsignificant. 


Table  3 


Gram  Analysis  and  Classification  Score 
Intercorrelations  for  Both  Types  of  Displays 


Display/Score 

Analysis 

Class. 

SS 

PS 

PM 

A 

Static  Lofargram 

Analysis 

Sound  source  (SS) 

1.00 

0.42* 

0.18 

0.04 

Predominant  spectrum  (PS) 

1.00 

0.15 

0.18 

Propulsion  mode  (PM) 

1.00 

0.28 

Classification 

Accuracy  (A) 

1.00 

OL-S2  Simulator: 

Analysis 

Sound  source  (SS) 

1.00 

0.52* 

0.57** 

0.60** 

Predominant  spectrum  (PS) 

1.00 

0.30 

0.41* 

Propulsion  mode  (PM) 

1.00 

0.93** 

Classification 

Accuracy  (A) 

1.00 

*p  <  .05. 

**p  <  .01. 


Table  4 

Correlations  Between  Display  Types  for  Gram  Analysis 
and  Classification  Scores 


Static  Lofargram 


OL-82  Simulator 

SS 

PS 

PM 

A 

Sound  source  (SS) 

.21 

.13 

.24 

-.08 

Predominant  spectrum  (PS) 

.09 

-.07 

.07 

.07 

Propulsion  mode  (PM) 

.06 

.28 

.28 

-.08 

Classification  accuracy  (A) 

.16 

.21 

.33 

.07 

DISCUSSION  AND  CONCLUSIONS 


Subjects  performed  acoustic  analysis  procedures  better  on  the  OL-82  sonar  system 
simulator  than  on  the  static  lofargrams,  probably  because  of  the  difference  in  the  number 
and  kinds  of  job  aids  available.  The  OL-82  simulator  provides  the  subjects  with  numerous 
electronic  job  aids,  computerized  computational  programs,  and  multiple  display  formats 
that  are  designed  to  make  the  operator's  job  of  acoustic  analysis  easier  and  more 
accurate.  With  the  lofargram  test,  subjects  were  limited  to  hand-held  calculators  and 
mechanical  dividers. 

Subjects'  ability  to  classify  targets  did  not  significantly  differ  for  the  two  display 
modes.  On  the  OL-82  test,  they  were  able  to  perform  the  analysis  and  classification  tasks 
at  about  the  same  level  of  proficiency.  However,  on  the  static  lofargram  test,  they 
performed  relatively  poorly  on  the  analysis  tasks,  but  were  able  to  classify  the  targets 
with  a  relatively  high  degree  of  accuracy.  It  may  be  that  they  were  able  to  classify 
targets  on  the  static  lofargrams  that  they  had  difficulty  analyzing  (i.e.,  they  were  unable 
to  show  how  they  analyzed  the  gram  to  reach  the  classification  decision)  because  of  their 
familiarity  with  the  specific  lofargrams  selected  for  the  test.  These  grams  were  obtained 
from  the  gram  library  provided  by  the  Acoustic  Analysis  School,  where  subjects  had 
received  their  basic  acoustic  analysis  training  and  had  studied  most  or  all  of  these  grams. 
In  discussions  following  the  static  lofargram  test,  subjects  reported  recognizing  the  target 
patterns  and  making  their  classifications  from  memory.  Thus,  it  appears  that  the 
classification  scores  obtained  on  this  test  may  not  be  a  good  general  indication  of  their 
ability  to  classify  unfamiliar  targets  using  the  static  lofargram. 

It  appears  that  tests  that  use  static  lofargrams  to  assess  OL-82  sonar  system  operator 
performance  may  underestimate  operator  proficiency  in  acoustic  analysis,  at  least  for 
sonar  operators  in  training.  Apparently,  it  is  easier  to  analyze  acoustic  information  on 
the  OL-82  sonar  system  than  on  static  lofargrams  because  of  the  number  and  kinds  of  job 
aids  available  to  the  operator.  Tests  that  measure  performance  using  static  grams  may 
reflect  the  increased  difficulty  of  that  largely  unaided  format.  In  addition,  because  the 
skills  required  to  perform  in  the  testing  situation  do  not  accurately  represent  the  skills 
required  to  perform  the  task  on-the-job,  the  operators  must  spend  time  practicing  skills 
only  used  in  the  testing  condition.  It  may  be  that  differences  shown  in  the  performance  of 
acoustic  analysis  procedures  on  the  two  display  formats  decrease  with  increased 
experience  level  of  the  operators.  However,  it  is  also  possible  that  time  spent  practicing 
test-oriented  skills  may  interfere  with  the  maintenance  of  job-oriented  skills  and  could 
potentially  degrade  job  performance  on  the  OL-82  sonar  system. 


RECOMMENDATIONS 

1.  The  validity  of  assessing  the  operational  readiness  of  sonar  operators  in  the  fleet 
should  be  determined  using  tests  that  rely  on  static  lofargrams  to  measure  acoustic 
analysis  and  dassificaiton  skills  that  are  operationally  performed  on  computer-based 
sonar  systems. 

2.  Research  should  be  conducted  to  determine  whether  on-the-job  performance  of 
acoustic  analysis  on  computer-based  sonar  systems  is  degraded  by  use  of  static  grams  in 
training,  in  promoting  skill  maintenance,  and  in  estimating  proficiency  on  tests. 

3.  Tests  of  operator  proficiency  that  use  static  lofargrams  should  measure  analysis 
and  classification  skills  on  lofargrams  not  previously  encountered  by  the  operator  in  the 
training  environment. 
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